Repository navigation
feat(io)!: zero-copy send for large writes on io_uring - #152
Merged
Merged
Conversation
`Write::write_all_owned` writes gathered buffers and hands them back only when the writer is done with them. On Linux over io_uring, a compio TCP stream sends a write of at least `ZeroCopySend::threshold` bytes with `IORING_OP_SENDMSG_ZC` and `MSG_WAITALL`: the call returns at the send's own completion, and a runtime task holds the operation and its buffers until the kernel's notification, which drops them. Unacknowledged zero-copy data is bounded by the socket's send buffer, which counts zero-copy pages. A send the kernel refuses for `RLIMIT_MEMLOCK` (`ENOMEM`, nothing sent) copies instead. Unix-domain streams, other drivers and other platforms copy as before; kernel TLS never reaches the zero-copy path, since its writer only issues copying writes, which TLS's own sendmsg requires (it rejects `MSG_ZEROCOPY` with `EOPNOTSUPP`). `ZeroCopySend` is the runtime-adjustable threshold, shared by every clone across threads; the default, 10 KiB, is where the kernel's MSG_ZEROCOPY documentation finds zero-copy pays off. `Compio` now carries it into every listener and stream it creates, so it is no longer a unit struct, and `Compio::tcp_stream` replaces `From<compio::net::TcpStream>` for adopting a stream.
A message's head and data frames go to the writer as owned buffers, so on io_uring a large flush is sent zero-copy; the buffer list's allocation is reused only when the writer hands it back.
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Linux, a compio TCP write at or above a threshold now goes out with
IORING_OP_SENDMSG_ZC. The call returns once the kernel has taken the send. The buffers stay held by a runtime task until the notification completion, then are dropped. Smaller writes still copy.Write::write_all_ownedwrites gathered buffers in full. It returns them when it copied and keeps them when the kernel holds them. Its default implementation copies, so tokio, async-io and the TLS writers are unchanged. h1's outbox now sends through it.compio::ZeroCopySendis the threshold, a shared runtime setting. A change applies to the next write on every existing stream. It defaults to 10 KiB, the sizemsg_zerocopy.rstnames as where zero-copy starts paying. 0 sends every eligible write zero-copy, andusize::MAXturns it off.Compionow carries the setting and so is no longerCopy. UseCompio::default()orCompio::with_zero_copy(..). Streams are adopted throughCompio::tcp_stream.There is no cap on how many sends a stream can have pending. Zero-copy pages still count toward
sk_wmem_queued, so the socket send buffer bounds them as it bounds copied data. Kernel TLS never reaches this path: the kTLS writer only copies, and the kernel's TLS sendmsg rejectsMSG_ZEROCOPYwith EOPNOTSUPP anyway. A send whose pages would exceedRLIMIT_MEMLOCKfails with ENOMEM before sending anything, and the rest of that write copies. A server that wants zero-copy under load needsCAP_IPC_LOCKor a raised limit.New tests on Linux check that the notification releases the buffer while the peer is stalled, and that changing the threshold switches existing streams. The io and h1 suites pass. The kTLS suites need CI.
Closes #126